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Abstract. The assembly and operation of the ISS has generated significant challenges 
that have ultimately impacted resources available to the program's primary mission: 
research. To address this, program personnel routinely perform trade-off studies on 
alternative options to enhance research. The approach, content, level of analysis and 
resulting outputs of these studies vary due to many factors, however, complicating the 
Program Manager's job of selecting the best option. To address this, the program 
requested a framework be developed to evaluate multiple research-enhancing options 
in a thorough, disciplined and repeatable manner, and to identify the best option on the 
basis of cost, benefit, and risk. The resulting framework consisted of a systematic 
methodology and a decision-support toolset. The framework provides quantifiable 
and repeatable means for ranking research-enhancing options for the complex and 
multiple-constraint domain of the space research laboratory. This paper describes the 
development, verification and validation of this framework and provides observations 
on its operational use. 
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Summary 


The assembly and operation of the International Space Station has generated significant 
unexpected challenges. These challenges have ultimately impacted resources available 
to the program's primary mission: research. To address these challenges, program 
personnel routinely perform technical trade-off studies on alternative options to 
enhance research. The approach, content, level of analysis and resulting outputs of 
these studies vary due to many factors, however, complicating the Program Manager's 
job of selecting the best option. 

The program requested a methodology be developed to evaluate multiple research- 
enhancing options in a thorough, disciplined and repeatable manner, and to identify the 
best option on the basis of cost, benefit, and risk. To this end a NASA civil- 
service /contractor composite team was assembled, and the task was conducted in three 
phases. 

During Phase I of the task, a new research capability enhancement decision-support 
paradigm comprising a systematic methodology and decision-support tool was created 
and demonstrated for the ISS Program Manager. In Phase II, the methodology and 
toolset were employed to identify and define options to enhance research onboard the 
ISS, and to evaluate the usefulness of the methodology and tools for future assessments 
by the ISS program. Finally, in Phase III, the toolset was refined to incorporate 
requested upgrades, and actual options under consideration by the program were 
analyzed to determine the most beneficial option on the basis of cost, benefit, and risk. 
The methodology and toolset were then handed off to ISS program personnel for future 
use. 

At the heart of the task was a closed-loop methodology developed by the team. The 
objective of the methodology was to improve organizational focus and increase the 
emphasis of studies on potential on-orbit research productivity. The methodology 
comprised four major stages: strategic, tactical, implementation, and review. In the 
strategic and tactical stages, statistical analyses of representative time periods in the 
distant and near future were performed to assess potential benefits of research 
enhancement options. During the implementation phase, the program office performed 
the normal functions of defining and executing a project plan (including 
implementation of selected research enhancement options) that addressed 
requirements, budget, schedule, risk management, and an external review process. The 
review phase provided feedback to either validate, or drive adjustment to, the strategy. 
A database was created to house study results and to minimize duplication of study 
efforts. 
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Supplementing the methodology was a purpose-built, spreadsheet-based tool. The 
Tool for Research Enhancement Decision Support (TREDS) was developed during 
Phase I and refined during the subsequent phases. The Analytical Hierarchy Process 
(AHP) documented in (Saaty 1983), was incorporated in TREDS to blend objective and 
subjective information toward summarizing the relative research merit of various 
options to the Program baseline. Objective information included three metrics chosen 
for their ability to represent research throughput, option costs, and option risks. 
Subjective information consisted of weightings the tool operator could apply to the 
three metrics to emphasize or de-emphasize their importance. 

Problem 

The International Space Station (ISS) now being assembled is designed to be a world- 
class research facility. The ISS is the largest volume of long-duration, low-Earth-orbit, 
human-tended, multidisciplinary experimental environment, ever to be offered. 
Research opportunities contributing to the resolution of technological challenges on 
Earth and enabling humankind's departure from our solar system are becoming 
available as never before. Even during assembly, research operations are being 
conducted using steadily expanding research capabilities. 

Assembling and putting into operation a spacecraft as complex as the ISS has generated 
significant unexpected program challenges. These challenges have ultimately impacted 
resources for the program's research mission. In the meantime, technical trade-off 
studies on alternative solutions that could mitigate this threat have continued; for 
example, the on-orbit crew time resource has been the subject of many such studies. 
The ISS Program Manager must evaluate the results of these alternative solution studies 
to select courses of action that are both technically and fiscally sound and defendable in 
the largest government policy framework. The fist step in this work was formulating a 
set of potential alternatives to enhance the research potential of the ISS. A summary of 
this work is shown in Figure 1. 
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Figure 1. Research Enhancement Alternatives 


The team then proceeded to demonstrate for the ISS Program Manager a new research 
capability enhancement decision-support framework comprising two components: a 
systematic, closed-loop methodology; and a complementary decision-support software 
tool. 


Methodology 


The objective of the proposed methodology was to improve organizational focus and 
increase studies emphasis on potential on-orbit research productivity. It comprised 
four major phases: strategy, tactical, implementation, and review. In the strategic 
phase, a statistical analysis of a representative ISS increment 4-6 years in the future 
would be performed to assess the research enhancement potential of alternatives during 
this time frame. The tactical phase would analyze specific increments 1-2 years in the 
future using actual payload planning data. During the implementation phase, the ISS 
Program Office (ISSPO) would perform its normal functions of defining and executing a 
project plan that addressed requirements, budget, schedule, risk management, and an 
external review process. It is during this phase that research enhancement options are 
implemented and realized research enhancement is measured. The review phase 
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would provide the feedback that would either validate or adjust the strategy. This 
methodology is shown in Figure 2. 
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Figure 2. Research Enhancement Methodology 


Supporting Toolset 

The Tool for Research Enhancement Decision Support-Strategic or Tactical (TREDS-S or 
TREDS-T) was created to compare alternative options by calculating a research 
capability figure of merit (FOM) for each option. TREDS did this by using a modified 
form of the time-tested Analytical Hierarchy Process (AHP). AHP seemed appropriate 
because both objective and subjective inputs needed to be considered, and AHP had 
been developed specifically for this type of problem. The subjective value weightings of 
three benefits of an option (research diversity per research quarter; hours of research 
per research quarter; and number of research facilities operable per research quarter) 
and the quantities of these benefits per option, as forecast by the Program's statistical 
simulations model, were converted into normalized coefficients. The individual 
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contributions of each option were then multiplied by these coefficients and summed to 
arrive at a single research return FOM for each option. 

The research capability figure of merit is expressed mathematically by the following 
equation: 

Research Capability FOM = 

C RD * W RD * (Research Diversity) + C RH * W RH * (Hours of Research) + C RF * W RF * (Research 
Facilities Operable), 

Where, 

- Cxx are the benefit coefficients relating the benefit metrics to the current 
baseline research capability 

- W X x are the normalized weightings of preference for each metric, calculated by 
two matrix operations inherent to AHP (0.09 <= W xx <= 0.81 and W RD + W RH + 

Wrp = 1.0). 

The FOM was then used to calculate an Additional Research Multiple and a Cumulative 
Return, which were made the basis of TREDS standard output and defined as follows: 

- The Additional Research Multiple was the FOM for a particular increment and 
represented the increase in research capability for an alternative. For example, a 
multiple of 1.2 indicated a research capability of 1.2 times the baseline, or a forecast 
increase in research capability of 20%. 

- Cumulative Return was the summation of the FOM over a given time period. 
For example, a cumulative research increase of 20% indicated that a particular 
alternative provided an increase in research capability of 20% over the baseline for the 
total time period under study. The baseline represents the research capability of the 
program if none of the research enhancement options are implemented. 

The team also recommended the chartering of an "ISS Study Manager" to coordinate 
the methodology's technical, schedule, risk, cost, and AHP assessment components. 
Finally, the Phase I team recommended the development and initialization of a 
Recommendations & Analyses Database. The R&A Database would be under the Study 
Manager's direct control and serve as the central studies reference database for the ISS 
Program. 

With these concepts preliminarily developed and demonstrated, the stage was set for 
their practical application. 
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Study Conduct 


The top-level approach to perform the studies is as follows: The study team would 
consider the various options appropriate to the study and advise the Study Manager, 
who would make the final selections. The Study Manager would then guide the team 
in obtaining the required data. Finally, the updated TREDS tool, in combination with 
the previously gathered data, would be used to rank-order the options based on their 
benefits, costs, and risks. The following paragraphs provide the main steps in the 
completions of a typical study 


A) BASELINE DEFINITION. The main output of the TREDS tool is a 
representation of the increase in research return provided by each option as a 
percentage increase over baseline capability. To arrive at these figures, it was 
necessary to define the baseline research capability of the ISS for each time period under 
study. Efforts were directed toward identifying the baseline research capabilities 
provided by the ISS during the study timeframe. This required the definition of the ISS 
configuration during that timeframe to include the station elements, laboratory space 
and capabilities, key operations expected, allocations of crew time, frequency of re- 
supply vehicles and up-mass capabilities. 

B) SELECTION AND DEFINITION OF ALTERNATIVE OPTIONS. Options to 
be studied were drawn from those studied previously, and selected by the Study 
Manager on the basis of two criteria: 

Perceived level of benefit 

Availability and maturity of benefit information. 

The total number of options to be studied in a typical study is around five. This 
number of options keeps the effort required at a reasonable level while still providing 
sufficient breadth of alternatives. The TREDS tool is capable of evaluating up to ten 
options at one time. 

C) DEFINITION OF OPTION BENEFITS. Options provide benefit to research 
by both directly and indirectly increasing the three research metrics: Research Diversity, 
Number of Research Facilities Operable, and the Hours of Research. All three metrics 
are based on a 90 day Research Quarter (RQ). Research Facilities Operable and Hours 
of Research are outputs of the Monte Carlo-based statistical model employed by the 
program, called the Payload Utilization Modeler (PLUM) tool. The PLUM operator 
calculates Research Diversity separately using information provided by the PLUM tool. 
These three metrics form the basis of the TREDS tool input. Option benefits should not 
be confused with option resources, which are the raw capabilities provided /required 
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by the options. Examples of resources include space shuttle mid-deck locker space and 
electrical power required. 

D) DEFINITION OF OPTION COSTS. In every study, both developmental and 
operational costs were obtained for each option. It is important to include both 
categories, as the composition of option costs can range from entirely operational to 
almost entirely developmental. To facilitate the gathering of cost information and 
provide consistency in the financial data reported, a representative from the program 
business management office was identified. All costs used in the studies reflect an 
escalation factor to account for inflation. 

E ) DEFINITION OF OPTION RISKS. Risk assessment was performed by the 
study team and certified by the program risk manager. A modified version of the ISS 
Program risk matrix is used to disclose the probability and consequence scores for risks 
assessed for each option. Modifications consisted of establishing likelihood and 
consequence ranges and thresholds for risks affecting the implementation of the 
research enhancement options and their impact to the research enhancement potential. 
This facilitates mathematical inclusion of identified risks in the calculation of research 
return. Risks are assessed in four categories: 

Technical implementation risks 

Schedule implementation risks 

Cost risks 

Increased safety risk caused by the implementation of each option. 

The TREDS tool can accept either likelihood scores directly, as percentages, in addition 
to accepting them in the form of levels 1 through 5. This was done to accommodate risk 
likelihoods as they are produced by the ISS program's Probabilistic Risk Assessment 
(PRA) capability. 

F) PERFORMANCE OF ANALYSES. Once data collection was complete, the 
TREDS runs were performed. The use of TREDS is straightforward since the tool is 
based on Microsoft Excel. Results of the TREDS outputs are summarized below. 

The primary outputs of TREDS are charts showing the cumulative return of the options 
over the study timeframe (relative to baseline) versus their cost. Two variations of these 
charts are available; one shows option costs in absolute dollars, while the other shows 
option costs as a percentage of total ISS program costs (Figure 3) . Risk penalties are 
assessed against the projected benefit and cost for the options. The benefit penalty is 
assessed based on the level of technical and schedule risks that were determined and 
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scored for each option. These penalties are discounted from the projected benefit using 
specialized algorithms developed for this purpose and are represented in the figure as a 
vertical line extending downward from the cost benefit point. The cost risk for each 
option is assessed as a percentage range around the cost estimate and is represented as 
a horizontal line crossing the cost benefit point. 
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Figure 3. TREDS Summary Output 

The option with the most favorable benefit to cost ratio can be identified by 
superimposing a line over the Y-axis and rotating it in a clockwise direction. The first 
option crossed by the rotating line is the most favorable. Using this technique on Figure 
3 identifies Option 1 as having the most favorable benefit to cost ratio. Option 1 also 
shows relatively little performance and schedule risk penalty. Option 3 is a distant 
second, and Option 3 offers little more benefit than Option 2 at a substantially higher 
cost. Both Option 2 and 3 have the potential for substantial performance and schedule 
risk penalties as well as a greater cost risk. These two options do, however, offer benefit 
in excess of program requirements. Option 1 falls well short of these requirements. 

A second output is the research increase vs. time chart, shown in Figure 4. This output 
was created to graphically demonstrate the phasing of option returns. In the example 
shown in Figure 4, two baselines are included ("US Core" and "IP Core") and the start 
dates of the options varied. It should be noted that in Figure 4, increases in research 
return provided by the options are shown as a research multiple, i.e. the net increase in 
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research capability provided by the option at a given time interval (as opposed to the 
cumulative returns shown in Figure 3). This information is very useful in determining 
the optimal time to phase-in or to discontinue a given option. For example. Option 1 in 
Figure 4 provides research capability enhancement of around 90% as compared to the 
baseline during 2005-2007 time frame. After 2007, the baseline configuration changes 
and this option will now provide a lesser return of about 50 % over the baseline. The 
decision maker will have to reconsider the total benefit-cost-risk case for this option 
after 2007 to determine if implementation is still beneficial to the program. 


Research Increase vs Time 



Figure 4. TREDS Research Multiple vs. Time 

Figure 5 shows another output of TREDS that is used to compare the relative level of 
risks between the options. An important component of this graphic is the 
representation of the relative level of safety risk inherent between options. This is an 
important element, for obvious reasons, in discriminating between alternative options 
This graphic provides another dimension for the decision maker in terms of risk since 
the risk information provided in Figure 3 is mainly based on assessing penalties on 
projected research enhancement potential. 

Other graphical outputs are also available such as the cost breakdown by fiscal year 
shown to allow a side-by-side comparison of the costs of each option. Cost spikes, 
timing of costs and cost escalation over time are readily apparent. This information is 
important for comparison and reconciliation with the available funding profile for the 
Program. 
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Option Risk Summary 



Figure 5. TREDS options risk summary 


Observations 

After the conduct of two full studies to verify and validate the framework, the study 
team found that the methodology and TREDS add value to ISS program decision- 
making. Specifically, the methodology: 

Provides a disciplined approach. 

Lends consistency to the evaluation of different options. 

Brings together the fundamental decision components: cost, benefit, and risk. 

- Is mathematical, repeatable, and comprehensive; detailed quantitative 
approaches are infused at an elementary level and documented. 

Provides a uniform, visible, and documented framework for decision 
advocacy both inside and outside the Program. 

Improves communication and integration among the different disciplines 
necessary to the studies process: business management, performance 

assessment teams, risk management, research planning, operations & 
integration, and technical study leads. 

Allows manipulation to examine sensitivities and investigate what-if 
scenarios. 
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- Allows comparisons and helps focus on what should be studied in detail. 

The methodology does, however, have certain limitations: 

It does not give a final answer in all cases. Option optimization will still be 
necessary to obtain the complete picture. Unknowns or poor definitions can 
drive the assumptions, and results can vary significantly based on those 
assumptions and on the fidelity of data. The Program Manager should use 
the results provided by the TREDS tool as an input to decision making. 

- Implementation of the methodology requires an investment. The increased 
level of rigor in the definition and analysis of the alternatives may require 
additional expenditure of resources such as manpower and computer time. 

Conclusion 

Prior to development and operational use of this framework, the ISS Program Manager 
was provided limited and dissimilar information in which to base programmatic 
decisions concerning the utilization potential of the ISS. Limitations included the need 
to develop a quantitative measure of research potential to perform cost benefit trades, 
the need to adjust the assessed benefit based on implementations risks, and the need to 
improve the systematic assessment of costs. The framework described in this paper 
addressed all of these limitations and provided an integrated system to perform 
feasibility studies to evaluate and rank research enhancement options. This framework 
is a fine example of systems engineering practice to improve the execution of a complex 
operational program. 
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